Antenna Season Report Notebook¶

Josh Dillon, Last Revised January 2022

This notebook examines an individual antenna's performance over a whole season. This notebook parses information from each nightly rtp_summarynotebook (as saved to .csvs) and builds a table describing antenna performance. It also reproduces per-antenna plots from each auto_metrics notebook pertinent to the specific antenna.

In [1]:
import os
from IPython.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
In [2]:
# If you want to run this notebook locally, copy the output of the next cell into the next line of this cell.
# antenna = "004"
# csv_folder = '/lustre/aoc/projects/hera/H5C/H5C_Notebooks/_rtp_summary_'
# auto_metrics_folder = '/lustre/aoc/projects/hera/H5C/H5C_Notebooks/auto_metrics_inspect'
# os.environ["ANTENNA"] = antenna
# os.environ["CSV_FOLDER"] = csv_folder
# os.environ["AUTO_METRICS_FOLDER"] = auto_metrics_folder
In [3]:
# Use environment variables to figure out path to the csvs and auto_metrics
antenna = str(int(os.environ["ANTENNA"]))
csv_folder = os.environ["CSV_FOLDER"]
auto_metrics_folder = os.environ["AUTO_METRICS_FOLDER"]
print(f'antenna = "{antenna}"')
print(f'csv_folder = "{csv_folder}"')
print(f'auto_metrics_folder = "{auto_metrics_folder}"')
antenna = "47"
csv_folder = "/home/obs/src/H6C_Notebooks/_rtp_summary_"
auto_metrics_folder = "/home/obs/src/H6C_Notebooks/auto_metrics_inspect"
In [4]:
display(HTML(f'<h1 style=font-size:50px><u>Antenna {antenna} Report</u><p></p></h1>'))

Antenna 47 Report

In [5]:
import numpy as np
import pandas as pd
pd.set_option('display.max_rows', 1000)
import glob
import re
from hera_notebook_templates.utils import status_colors, Antenna
In [6]:
# load csvs and auto_metrics htmls in reverse chronological order
csvs = sorted(glob.glob(os.path.join(csv_folder, 'rtp_summary_table*.csv')))[::-1]
print(f'Found {len(csvs)} csvs in {csv_folder}')
auto_metric_htmls = sorted(glob.glob(auto_metrics_folder + '/auto_metrics_inspect_*.html'))[::-1]
print(f'Found {len(auto_metric_htmls)} auto_metrics notebooks in {auto_metrics_folder}')
Found 15 csvs in /home/obs/src/H6C_Notebooks/_rtp_summary_
Found 15 auto_metrics notebooks in /home/obs/src/H6C_Notebooks/auto_metrics_inspect
In [7]:
# Per-season options
mean_round_modz_cut = 4
dead_cut = 0.4
crossed_cut = 0.0

def jd_to_summary_url(jd):
    return f'https://htmlpreview.github.io/?https://github.com/HERA-Team/H6C_Notebooks/blob/main/_rtp_summary_/rtp_summary_{jd}.html'

def jd_to_auto_metrics_url(jd):
    return f'https://htmlpreview.github.io/?https://github.com/HERA-Team/H6C_Notebooks/blob/main/auto_metrics_inspect/auto_metrics_inspect_{jd}.html'

Load relevant info from summary CSVs¶

In [8]:
this_antenna = None
jds = []

# parse information about antennas and nodes
for csv in csvs:
    df = pd.read_csv(csv)
    for n in range(len(df)):
        # Add this day to the antenna
        row = df.loc[n]
        if isinstance(row['Ant'], str) and '<a href' in row['Ant']:
            antnum = int(row['Ant'].split('</a>')[0].split('>')[-1]) # it's a link, extract antnum
        else:
            antnum = int(row['Ant'])
        if antnum != int(antenna):
            continue
        
        if np.issubdtype(type(row['Node']), np.integer):
            row['Node'] = str(row['Node'])
        if type(row['Node']) == str and row['Node'].isnumeric():
            row['Node'] = 'N' + ('0' if len(row['Node']) == 1 else '') + row['Node']
            
        if this_antenna is None:
            this_antenna = Antenna(row['Ant'], row['Node'])
        jd = [int(s) for s in re.split('_|\.', csv) if s.isdigit()][-1]
        jds.append(jd)
        this_antenna.add_day(jd, row)
        break
In [9]:
# build dataframe
to_show = {'JDs': [f'<a href="{jd_to_summary_url(jd)}" target="_blank">{jd}</a>' for jd in jds]}
to_show['A Priori Status'] = [this_antenna.statuses[jd] for jd in jds]

df = pd.DataFrame(to_show)

# create bar chart columns for flagging percentages:
bar_cols = {}
bar_cols['Auto Metrics Flags'] = [this_antenna.auto_flags[jd] for jd in jds]
bar_cols[f'Dead Fraction in Ant Metrics (Jee)'] = [this_antenna.dead_flags_Jee[jd] for jd in jds]
bar_cols[f'Dead Fraction in Ant Metrics (Jnn)'] = [this_antenna.dead_flags_Jnn[jd] for jd in jds]
bar_cols['Crossed Fraction in Ant Metrics'] = [this_antenna.crossed_flags[jd] for jd in jds]
bar_cols['Flag Fraction Before Redcal'] = [this_antenna.flags_before_redcal[jd] for jd in jds]
bar_cols['Flagged By Redcal chi^2 Fraction'] = [this_antenna.redcal_flags[jd] for jd in jds]
for col in bar_cols:
    df[col] = bar_cols[col]

z_score_cols = {}
z_score_cols['ee Shape Modified Z-Score'] = [this_antenna.ee_shape_zs[jd] for jd in jds]
z_score_cols['nn Shape Modified Z-Score'] = [this_antenna.nn_shape_zs[jd] for jd in jds]
z_score_cols['ee Power Modified Z-Score'] = [this_antenna.ee_power_zs[jd] for jd in jds]
z_score_cols['nn Power Modified Z-Score'] = [this_antenna.nn_power_zs[jd] for jd in jds]
z_score_cols['ee Temporal Variability Modified Z-Score'] = [this_antenna.ee_temp_var_zs[jd] for jd in jds]
z_score_cols['nn Temporal Variability Modified Z-Score'] = [this_antenna.nn_temp_var_zs[jd] for jd in jds]
z_score_cols['ee Temporal Discontinuties Modified Z-Score'] = [this_antenna.ee_temp_discon_zs[jd] for jd in jds]
z_score_cols['nn Temporal Discontinuties Modified Z-Score'] = [this_antenna.nn_temp_discon_zs[jd] for jd in jds]
for col in z_score_cols:
    df[col] = z_score_cols[col]

ant_metrics_cols = {}
ant_metrics_cols['Average Dead Ant Metric (Jee)'] = [this_antenna.Jee_dead_metrics[jd] for jd in jds]
ant_metrics_cols['Average Dead Ant Metric (Jnn)'] = [this_antenna.Jnn_dead_metrics[jd] for jd in jds]
ant_metrics_cols['Average Crossed Ant Metric'] = [this_antenna.crossed_metrics[jd] for jd in jds]
for col in ant_metrics_cols:
    df[col] = ant_metrics_cols[col]

redcal_cols = {}
redcal_cols['Median chi^2 Per Antenna (Jee)'] = [this_antenna.Jee_chisqs[jd] for jd in jds]
redcal_cols['Median chi^2 Per Antenna (Jnn)'] = [this_antenna.Jnn_chisqs[jd] for jd in jds]   
for col in redcal_cols:
    df[col] = redcal_cols[col]

# style dataframe
table = df.style.hide_index()\
          .applymap(lambda val: f'background-color: {status_colors[val]}' if val in status_colors else '', subset=['A Priori Status']) \
          .background_gradient(cmap='viridis', vmax=mean_round_modz_cut * 3, vmin=0, axis=None, subset=list(z_score_cols.keys())) \
          .background_gradient(cmap='bwr_r', vmin=dead_cut-.25, vmax=dead_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .background_gradient(cmap='bwr_r', vmin=crossed_cut-.25, vmax=crossed_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .background_gradient(cmap='plasma', vmax=4, vmin=1, axis=None, subset=list(redcal_cols.keys())) \
          .applymap(lambda val: 'font-weight: bold' if val < dead_cut else '', subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val < crossed_cut else '', subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val > mean_round_modz_cut else '', subset=list(z_score_cols.keys())) \
          .applymap(lambda val: 'color: red' if val > mean_round_modz_cut else '', subset=list(z_score_cols.keys())) \
          .bar(subset=list(bar_cols.keys()), vmin=0, vmax=1) \
          .format({col: '{:,.4f}'.format for col in z_score_cols}) \
          .format({col: '{:,.4f}'.format for col in ant_metrics_cols}) \
          .format('{:,.2%}', na_rep='-', subset=list(bar_cols.keys())) \
          .set_table_styles([dict(selector="th",props=[('max-width', f'70pt')])]) 

Table 1: Per-Night RTP Summary Info For This Atenna¶

This table reproduces each night's row for this antenna from the RTP Summary notebooks. For more info on the columns, see those notebooks, linked in the JD column.

In [10]:
display(HTML(f'<h2>Antenna {antenna}, Node {this_antenna.node}:</h2>'))
HTML(table.render(render_links=True, escape=False))

Antenna 47, Node N06:

Out[10]:
JDs A Priori Status Auto Metrics Flags Dead Fraction in Ant Metrics (Jee) Dead Fraction in Ant Metrics (Jnn) Crossed Fraction in Ant Metrics Flag Fraction Before Redcal Flagged By Redcal chi^2 Fraction ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score Average Dead Ant Metric (Jee) Average Dead Ant Metric (Jnn) Average Crossed Ant Metric Median chi^2 Per Antenna (Jee) Median chi^2 Per Antenna (Jnn)
2459990 not_connected 100.00% 100.00% 100.00% 0.00% - - 10.973387 13.983848 4.328170 4.425124 9.040557 10.939606 2.054434 -0.065345 0.0319 0.0556 0.0162 nan nan
2459989 not_connected 100.00% 100.00% 100.00% 0.00% - - 10.809785 14.178886 3.832185 4.168453 7.993833 9.175737 1.645247 0.111473 0.0313 0.0518 0.0142 nan nan
2459988 not_connected 100.00% 100.00% 100.00% 0.00% - - 12.941451 16.565190 4.427312 4.483980 10.768474 13.109728 1.493601 -0.033449 0.0304 0.0499 0.0139 nan nan
2459987 not_connected 100.00% 100.00% 100.00% 0.00% - - 10.918473 13.855115 4.421545 4.668217 6.408348 7.924481 5.151369 3.041934 0.0315 0.0543 0.0160 nan nan
2459986 not_connected 100.00% 100.00% 100.00% 0.00% - - 13.373326 16.994771 4.852378 4.955426 9.382890 11.177469 6.899817 9.568464 0.0307 0.0549 0.0167 nan nan
2459985 not_connected 100.00% 100.00% 100.00% 0.00% - - 12.438472 15.358945 4.537308 4.692170 7.215462 8.530995 3.685445 0.771001 0.0309 0.0529 0.0155 nan nan
2459984 not_connected 100.00% 100.00% 99.08% 0.00% - - 12.040574 14.687224 4.822874 4.974057 9.579906 12.154149 5.155234 3.860397 0.0302 0.0583 0.0194 nan nan
2459983 not_connected 100.00% 100.00% 100.00% 0.00% - - 11.435620 14.485115 4.425643 4.443143 9.281033 11.087571 4.941586 6.332835 0.0304 0.0559 0.0174 nan nan
2459982 not_connected 100.00% 100.00% 100.00% 0.00% - - 10.360046 11.444914 3.883626 3.984248 4.545601 5.226099 2.775300 3.281304 0.0310 0.0569 0.0177 nan nan
2459981 not_connected 100.00% 100.00% 100.00% 0.00% - - 10.699234 13.374602 4.545585 4.530156 10.482399 12.237636 1.958568 0.074408 0.0307 0.0544 0.0162 nan nan
2459980 not_connected 100.00% 100.00% 100.00% 0.00% - - 10.648032 13.011825 4.024846 4.267239 8.979993 10.713398 5.227227 5.137142 0.0307 0.0515 0.0140 nan nan
2459979 not_connected 100.00% 100.00% 100.00% 0.00% - - 10.980093 13.489564 3.550119 3.854167 8.925348 10.060514 1.289001 0.541748 0.0319 0.0463 0.0113 nan nan
2459978 not_connected 100.00% 100.00% 100.00% 0.00% - - 11.080897 13.699626 3.902974 4.146491 9.329700 10.909370 1.973065 1.140804 0.0294 0.0435 0.0102 nan nan
2459977 not_connected 100.00% 100.00% 100.00% 0.00% - - 11.344309 14.411134 4.046719 4.335552 9.435779 11.432257 4.015019 3.235159 0.0307 0.0520 0.0149 nan nan
2459976 not_connected 100.00% 100.00% 100.00% 0.00% - - 11.319698 13.942114 4.137126 4.323275 9.350754 10.740427 1.012800 0.272216 0.0305 0.0464 0.0112 nan nan

Load antenna metric spectra and waterfalls from auto_metrics notebooks.¶

In [11]:
htmls_to_display = []
for am_html in auto_metric_htmls:
    html_to_display = ''
    # read html into a list of lines
    with open(am_html) as f:
        lines = f.readlines()
    
    # find section with this antenna's metric plots and add to html_to_display
    jd = [int(s) for s in re.split('_|\.', am_html) if s.isdigit()][-1]
    try:
        section_start_line = lines.index(f'<h2>Antenna {antenna}: {jd}</h2>\n')
    except ValueError:
        continue
    html_to_display += lines[section_start_line].replace(str(jd), f'<a href="{jd_to_auto_metrics_url(jd)}" target="_blank">{jd}</a>')
    for line in lines[section_start_line + 1:]:
        html_to_display += line
        if '<hr' in line:
            htmls_to_display.append(html_to_display)
            break

Figure 1: Antenna autocorrelation metric spectra and waterfalls.¶

These figures are reproduced from auto_metrics notebooks. For more info on the specific plots and metrics, see those notebooks (linked at the JD). The most recent 100 days (at most) are shown.

In [12]:
for i, html_to_display in enumerate(htmls_to_display):
    if i == 100:
        break
    display(HTML(html_to_display))

Antenna 47: 2459990

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 13.983848 13.983848 10.973387 4.425124 4.328170 10.939606 9.040557 -0.065345 2.054434

Antenna 47: 2459989

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 14.178886 14.178886 10.809785 4.168453 3.832185 9.175737 7.993833 0.111473 1.645247

Antenna 47: 2459988

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 16.565190 16.565190 12.941451 4.483980 4.427312 13.109728 10.768474 -0.033449 1.493601

Antenna 47: 2459987

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 13.855115 10.918473 13.855115 4.421545 4.668217 6.408348 7.924481 5.151369 3.041934

Antenna 47: 2459986

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 16.994771 16.994771 13.373326 4.955426 4.852378 11.177469 9.382890 9.568464 6.899817

Antenna 47: 2459985

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 15.358945 15.358945 12.438472 4.692170 4.537308 8.530995 7.215462 0.771001 3.685445

Antenna 47: 2459984

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 14.687224 12.040574 14.687224 4.822874 4.974057 9.579906 12.154149 5.155234 3.860397

Antenna 47: 2459983

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 14.485115 11.435620 14.485115 4.425643 4.443143 9.281033 11.087571 4.941586 6.332835

Antenna 47: 2459982

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 11.444914 10.360046 11.444914 3.883626 3.984248 4.545601 5.226099 2.775300 3.281304

Antenna 47: 2459981

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 13.374602 13.374602 10.699234 4.530156 4.545585 12.237636 10.482399 0.074408 1.958568

Antenna 47: 2459980

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 13.011825 13.011825 10.648032 4.267239 4.024846 10.713398 8.979993 5.137142 5.227227

Antenna 47: 2459979

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 13.489564 10.980093 13.489564 3.550119 3.854167 8.925348 10.060514 1.289001 0.541748

Antenna 47: 2459978

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 13.699626 13.699626 11.080897 4.146491 3.902974 10.909370 9.329700 1.140804 1.973065

Antenna 47: 2459977

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 14.411134 11.344309 14.411134 4.046719 4.335552 9.435779 11.432257 4.015019 3.235159

Antenna 47: 2459976

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
47 N06 not_connected nn Shape 13.942114 13.942114 11.319698 4.323275 4.137126 10.740427 9.350754 0.272216 1.012800

In [ ]: